Tablets and touch screens have long been developed and are widely used in society. Typically these devices facilitate user input in conjunction with computer systems. Many handheld devices, such as the personal digital assistant (PDA), incorporate touch screen technology as a means of user interaction. There currently exist many patents relating to the design, manufacture, and use of tablets or touch screens, and representative USPTO examples relating to the proposed invention are given below:
U.S. Pat. No.Inventor(s)Title5,938,163Gotham et al.Articulating touchscreen interface5,956,020D'Amico et al.Touchscreen controller with pen and/orfinger inputs6,018,336Akiyama et al.Computer system having digitizing tablet6,738,049Kiser et al.Image based touchscreen device6,901,277Kaufman et al.Methods for generating a lung report6,317,617Gilhuijs et al.Method, computer program product, andsystem for the automated analysis oflesions in magnetic resonance, mammogramand ultrasound images6,204,201Xu et al.Computerized detection of lung nodulesusing energy-subtracted soft-tissue andstandard chest images6,234,979MerzenichComputerized method and device forremediating exaggerated sensory responsein an individual with an impaired sensorymodality6,315,569ZaltmanMetaphor elicitation technique withphysiological function monitoring
U.S. Pat. No. 5,938,163 to Gotham describes an articulating system for mounting a touch screen (control console) onto a machine. A swiveling base on top of a sliding platform allows the console to be positioned in a plurality of positions within a 3D space. The purpose of said patent is to allow ergonomic access to the console regardless of where the user is positioned.
U.S. Pat. No. 5,956,020 to D'Amico describes the specifics of a touch screen controller for enabling operation of a touch sensitive screen in response to commands from an application program executing on a data processor that is electronically coupled to the controller. Input to the screen may be by pen or finger. Additional claims relate to the ability of the controller to take various actions, including resizing of the active area, dependent on the state of the host computer.
U.S. Pat. No. 6,018,336 to Akiyama describes use of a digitizing tablet to emulate computer mouse function. Event data, such as tapping motions, are added to the coordinate values x and y and sent to the tablet driver. These data are interpreted by a pointing management system that then further structures the information, prior to communication with the computer's operating system. In this manner the many functions of the mouse can be simulated, including drag-and-drop, and left/right clicks.
U.S. Pat. No. 6,738,049 to Kiser describes a customizable input device comprising a display device, a touch screen, and a microcontroller. Claimed as part of this patent is a system whereby images are displayed to a user prompting some form of response via the touch screen. Input from the touch screen is interpreted by the microcontroller before being sent to a computer system. The computer then alters the display accordingly. Further claims include the ability to overlay a transparent touch screen directly above the image surface. The device is connected to the computer via universal serial bus (USB).
A number of representative patents relate use of tablet and touch screen technology with analysis of diagnostic medical images. U.S. Pat. No. 6,901,277 to Kaufman describes the software, methods and user interfaces for viewing and generating a lung report. Information regarding lung nodules can be stored and used to relocate the previously localized lung nodules in a second, follow-up imaging scan. Scans can be analyzed manually or automatically and statistics are produced for reporting about localized lung nodules based on changes between initial and follow-up scans. The images can come from various imaging modalities such as MRI or CT. Users interact with the system via various input peripherals.
Similarly, U.S. Pat. No. 6,317,617 to Gilhuijs describes a method and system for the computerized automatic analysis of lesions in magnetic resonance (MR) images, a computer programmed to implement the method, and a data structure for storing required parameters. Specifically, the system offers the ability to conduct computerized analysis of lesions in the breast using spatial, temporal and/or hybrid measures. The system also allows for the enhanced visualization of the breast and its pathological states. The system also includes an option to merge the extracted features with those from x-ray and/or ultrasound images to further characterize the lesion and/or make a diagnosis. Other imaging techniques, such as CT and/or MRI can also be employed.
U.S. Pat. No. 6,240,201 to Xu describes a method, system, and computer-readable medium configured for computerized detection of lung abnormalities. This includes the acquisition of various images of the chest and identifying abnormalities through a series of analyses, such as difference maps. As mentioned in the Kaufman and Gilhuijs patents, the images may be obtained using different imaging modalities. Control of the system is achieved using input devices, such as keyboards, mice, and/or computerized tablets.
Other prior art relates to use of tablets and touch screens for functional monitoring. U.S. Pat. No. 6,315,569 to Zaltman describes a process and apparatus for using a metaphor elicitation technique in conjunction with physiological function monitoring to elicit, organize and analyze data pertaining to a research topic. The metaphor elicitation technique process and apparatus is improved with the acquisition of data related to a user's physiological functioning. This data provides further insight and understanding which can be used in creating an appropriate marketing campaign for a product, improving inter-office communications and determining the presence of pre-existing biases or beliefs. Physiological monitoring may include functional magnetic resonance imaging, positron emission tomography, galvanic skin response or conductance, event related potentials, or heart rate changes. The technique involves a series of specific tasks, one of which, the Mental Map, involves image validation and creation using a mouse, cursor, or pressure sensitive digitizing tablet (via a stylus or a finger).
U.S. Pat. No. 6,234,979 to Merzenich outlines a method of using an apparatus for implementing a training regime having a stimulator and an input device, for remediating exaggerated responses associated with a super-group of neurons in an individual with an associated impaired modality. Many modifications to the method are mentioned, for example, the invention can be practiced with or without feedback. Furthermore this feedback may be manual or automated. Manual feedback can provide an indication that an input is causing discomfort or pain, or the individual is able to distinguish the stimuli. Examples of automated feedback include brain imaging such as MEG and fMRI to monitor changes and responses within the super-group of neurons. Different types of input devices, such as a computerized tablet, can be used to input commands and other instructions into the computer system, which serves as the backbone of the invention.
Though tablet devices have achieved considerable popularity and commercial success, there has been continuing demand to expand on their use and application. One such application is in the field of functional neuroimaging, which involves the measurement of brain activity associated with human behavior. Techniques such as functional magnetic resonance imaging (fMRI) allow researchers and clinicians to visualize regions of brain activity with high spatial resolutions (millimeters), in accordance to the tasks being performed by the subject inside the scanner.
Functional neuroimaging is a broad field encompassing a variety of medically-oriented applications, including use in neurosurgical planning, potential detection of neurodegenerative disease, and monitoring the response of neural tissue to targeted therapeutic interventions such as pharmacotherapy, stem cell therapy, cognitive rehabilitation, or physical rehabilitation. In their idealized forms none of the previously mentioned patents focus on augmenting functional neuroimaging through use of tablet devices for such applications.
Specifically, certain publications such as U.S. Pat. No. 6,901,277 to Kaufman, U.S. Pat. No. 6,317,617 to Gilhuijs and U.S. Pat. No. 6,240,201 to Xu, describe use of imaging technologies such as CT and MRI to acquire anatomical images of the body and describe various means to manipulate and analyze the data. Interaction with their respective systems is proposed through use of input peripherals, such as a mice, keyboards, and/or computerized tablets. These devices would typically be used by the CT or MRI technologist during image acquisition or by a radiologist during image interpretation. The proposed invention differs significantly from these previous works in its design and application, has been engineered primarily to operate within the bore of an MRI scanner, and is for use by the individual being scanned during functional neuroimaging, or more specifically functional magnetic resonance imaging (fMRI) of brain activity.
U.S. Pat. No. 6,315,569 (Zaltman) and U.S. Pat. No. 6,234,979 (Merzenich) concern techniques for probing and rehabilitating response to stimuli. The invention disclosed in U.S. Pat. No. 6,315,569 describes a procedure for evaluating responses to marketing material, whereas, U.S. Pat. No. 6,234,979 describes a training regime to alleviate exaggerated response to sensory stimuli (e.g. hypersensitivity to sound in autistic children). Both mention the use of adjunct physiological measurements such as PET and fMRI as feedback mechanisms and the use of input devices for interaction with a computer system. However, the merger of the two disparate ideas is neither discussed nor elaborated in any detail. Instead the peripherals are discussed in the context of a means for generic user input outside of a MR scanner environment. As mentioned above the present invention details a system specific for drawing and writing inside the MR scanner environment specifically to augment fMRI capabilities.
Functional MRI is a relatively new technology that relies on neurovascular coupling and hyperemia effects. Sensitive to small percentage changes in the flow, volume, and oxygenation of blood that occur over short time spans in the local vicinity of neurons that have become more electrically active in comparison to baseline levels, fMRI makes it possible to identify spatiotemporal patterns of activation in various regions of the brain associated with specific tasks, typically on the second and millimeter time and spatial scales, respectively.
Considering a combinations of factors such as cost, availability, risk and invasiveness, sensitivity, spatial and temporal resolution, and volume of coverage within the brain, fMRI has numerous advantages over other functional neuroimaging modalities. However, there are some technical challenges associated with this technology. For example, fMRI is typically conducted in a very strong static magnetic field (>1 Tesla), with accompanying weaker but dynamic spatially varying magnetic fields (˜10 mT/m gradients and ˜100 T/m/μs slew rates), and with stringent constraints regarding radiofrequency electromagnetic interference (EMI) from other nearby electronic devices. Therefore, electronic equipment designed for use within the scanner must adhere to these strong electromagnetic restrictions.
Functional MRI is also very sensitive to head motion. Very subtle head motion on the order of millimeters can significantly degrade image quality. As well, the behavioral tasks and analytic approaches that are adopted in fMRI studies greatly influence the resulting images of brain activity. Much work has already gone into the development of fMRI-compatible devices including MRI-compatible display systems, response pads, joysticks, and audio equipment. However, to date, no device has been manufactured or patented that would capture and display drawing movements within the scanner. This invention addresses this deficit through the design of the fMRI-compatible drawing tablet.
There is strong motivation to permit and record writing and drawing movements during fMRI. For example, the ability to record and display subject responses using a pen-like interface would allow traditional neuropsychological assessments to be performed inside the scanner. There are hundreds of neuropsychological tests for probing the various aspects of cognitive function, such as attention, memory, executive function, language, and general cognitive ability. The majority of these tests are pen-and-paper based in that the clinician or neuropsychologist presents the subject with a set of instructions and the subject reacts by writing or drawing their response on paper.
Examples of traditional pen-and-paper tests include the Rey-Osterrieth Complex Figure [(Rey, A., The psychological examination of cases of traumatic encephalopathy. Archives de Psychologie, 37:126-139, 1941) & (Osterrieth, P. A., Copying a complex figure: Contributions to the study of perception and memory. Archives de Psychologie, 30:203-353, 1944)], the Trail Making Test [Lezak et al., Neuropsychological assessment, 4th Edition. New York: Oxford University Press, 2004], the Clock Drawing Test [Sunderland, T. et al., Clock drawing in Alzheimer's disease. A novel measure of dementia severity. J Am Gedatr Soc, 37:725-9, 1989], various letter and symbol cancellation tasks, the Benton Visual Retention Test [Benton, A. L., The Revised Visual Retention Test: Clinical and Experimental Applications. New York: Psychological Corp, 1963] and the Goodenough-Harris Drawing Test [Harris, D. B., The Goodenough-Harris drawing test. Los Angeles: Harcourt-Jovanovich, 1963]. Each test is designed to probe a particular aspect of cognitive function and their utility in the clinic has been proven through rigorous research and clinical trials. However, the exact neurological underpinnings of these tests are often unknown or are based on lesion studies, which have their own inherent confounds.
It would be of great benefit to researchers and clinicians if these tests could be performed together with fMRI to identify the brain regions that are engaged. Knowing the areas of the brain implicated in such tests is critical for clinical application of the test and is especially important during planning of rehabilitation strategy or pharmacological intervention. To conduct these studies it is important that the tests be reproduced during fMRI in a manner similar to their administration in the clinic.
Similar to the neuropsychological tests mentioned above there are a multitude of personality tests and scales that require written responses. Some examples include the Montgomery-Asberg Depression Rating Scale (MADRS) [Montgomery, S. A., & Asberg, M., A new depression scale designed to be sensitive to change. Br J Psychiatry, 34:382-389, 1979], the Brief Psychiatric Rating Scale (BPRS) [Overall, J. E., & Gorham, D. R., The Brief Psychiatric Rating Scale. Psychological Report, 10:799-812, 1962], and the Self-Control Rating Scale [Kendall, P., & Wilcox, L., Self-control in children: Development of a rating scale. Journal of Consulting and Clinical Psychology, 47:1020-1029, 1979]. These measures are also largely understood in terms of their underlying functional networks so investigation with fMRI would be highly valued.
Besides investigation of neuropsychological tests and scales one domain of particular scientific interest is general drawing and copying behaviour. As such there are several published scientific papers that have examined the neural correlates of drawing and/or copying during functional neuroimaging. Most relevant are papers by Katanoda et al. [Katanoda et al., A functional MRI study on the neural substrates for writing. Hum Brain Mapp, 13(1): 34-42, 2001] and Makuuchi et al. [Makuuchi et al., Both parietal lobes are involved in drawing: a functional MRI study and implications for constructional apraxia. Cogn Brain Res, 16(3): 338-47, 2003]. Both studies help to provide additional knowledge critical in the treatment of brain-damaged patients incapable of performing drawing or copying tasks. A particular condition causing such a deficit is called constructional apraxia, and refers to the inability of patients to assemble the elements of a model object in their correct spatial relationships.
Further fMRI investigation into the exact neurological underpinnings of this condition is underway. However, to do so, the task should be executed in a manner similar to real life. Both studies described immediately above involved writing/drawing with the right index finger, either in the air or on a fixed surface, during fMRI. Such a configuration is very unlike how people write in everyday life and therefore the generalizability of the reported findings is questionable. Furthermore, without a method to capture user input (ie. what the user “writes”) it becomes difficult to assess the behavioral performance that underlies the resulting images of brain activity. An fMRI-compatible tablet system would help to overcome these limitations.
Additional studies have examined the task of writing/drawing with other functional brain mapping techniques. Examples include Bhattacharya et al., who in their 2005 paper [Bhattacharya et al., Drawing on mind's canvas: Differences in cortical integration patterns between artists and non-artists. Hum Brain Mapp, 26(1): 1-14, 2005] compared multi-channel electroencephalograph (EEG) signals in groups of artists and non-artists. The results of this study demonstrated that in artists, patterns of functional cooperation between cortical regions during mental creation of drawings were significantly different from those in non-artists. This study did not use any sort of drawing device, only mental imagery, but further demonstrates the level of interest in this particular field of neuroscience.
Another study [Siebner et al., Brain correlates of fast and slow handwriting in humans: a PET-performance correlation analysis. Eur J Neurosci. 14(4): 726-36, 2001] employed H215O positron emission tomography (PET) to measure the regional cerebral blood flow (rCBF) in 10 healthy subjects during handwriting. This particular study used standard pen-based recording, with and without feedback, to examine the cerebral control of velocity during handwriting. Because it was not limited by the confines and constraints of fMRI, use of a traditional writing tool was possible. However, PET is generally acknowledged to exhibit lower spatial and temporal resolution than fMRI, and additionally requires the administration of radiopharmaceuticals. The study concluded that there exists a set of regions particularly involved in the processing of slow closed-loop writing movements (i.e. without feedback and half of normal writing speed). These regions included the right lateral premotor cortex, the left anterior parietal cortex, the left anterior putamen, the left rostral supplementary motor area and the right precuneus.
Other relevant scientific work appeared as an abstract at the Annual Meeting of the Organization for Human Brain Mapping in 2004. [Reithler et al., Resistance-based Recording of Pen Trajectories in an fMRI setting, Hum Brain Mapp, Abst. #320, 2004]. The authors describe a system for capturing pen movements during fMRI using a resistive touch surface. However, the device operates very differently from the invention presented here. First, a single value is output from the device, enabling only a 1-dimensional measurement of movement. Therefore, the paths that users are asked to trace must be known a priori. Second, the basic working principle of the device is that the pen and path surface form an electrical circuit that exhibits resistance changes as the pen is moved around the drawing surface. Third, no visual feedback of performance is provided; captured drawing movements are not visible to the user. Although useful in some applications, this device is more limited in its ability to enable and record realistic copying and drawing behaviour than the present invention.
Beyond neuropsychological assessment and basic scientific investigations into drawing and copying, the fMRI-compatible tablet would also be useful for exploring writing rehabilitation strategies. The physiology of acquired disorders of writing and mechanisms for their recovery are largely unknown. These disorders are common in stroke patients who often suffer some form paresis in their limbs (hand) as a result of the stroke.
There has been a great deal of research exploring post-stroke of motor function; however few have directly investigated the task of writing. One exception is a paper by Papathanassiou et al, titled “Plasticity of motor cortex excitability induced by rehabilitation therapy for writing”, in which the authors study the brain's ability to reorganize neural pathways (i.e. plasticity) during a rehabilitative writing therapy. Using transcranial magnetic stimulation (TMS) and an electroencephalogram (EEG) the authors discovered that rehabilitation aimed to increase the use of the paretic hand during writing may induce recruitment of previously silent neural pathways even in poorly recovered post-stroke patients. This is a considerable finding and illustrates the importance of employing such strategies during a rehabilitation program. With this in mind, further research into which writing regimes are most beneficial and to what degree they are able to reorganize the brain must be pursued. Functional MRI would be the favoured technology to conduct these studies due to its excellent spatial resolution and imaging characteristics, but an fMRI-compatible writing tablet requires development.
U.S. Pat. No. 6,234,979 to Merzenich discusses a remediation technique for individuals with an impaired sensory modality but is unspecific to the task of writing. Furthermore, brain imaging is limited to use as a feedback mechanism during their intervention and not as a means to investigate neuronal plasticity and reorganization in the brain.
Use of the fMRI-compatible tablet extends beyond established neuropsychological assessment and rehabilitation techniques and provides increased flexibility for designing new behavioural tasks. For example, the device could be used as a generic input peripheral, similar to a mouse or keyboard but specific for use within the MRI bore during fMRI. This is important because much of fMRI usage involves (but is also limited by) use of fMRI-compatible “response boxes” that enable the user to respond by pressing a button. Such response boxes are sufficient for simple user input, for example choosing one response by button press from a small list based on some previously presented stimuli. However, these devices typically only offer one button press per finger, for several fingers. Increasing the complexity of such devices (e.g. more buttons) becomes a “response mapping” problem requiring potentially significant training time for individuals to become comfortable learning to make selections with a non-intuitive interface.
Furthermore, currently there is no simple way for the user to provide a response at a specific x-y position during fMRI, especially to the precision and accuracy achievable with a mouse or tablet. Several fMRI-compatible joysticks are available but these require buttons for response selection. Mice have their own inherent limitations because they require a smooth, flat surface for operation and thus are ill-suited to the confines of an MR bore. A computerized tablet replicates much of the functionality of a mouse but offers a more intuitive interface, especially in the context of fMRI. This functionality enables manipulation of various graphical user interface elements such as check boxes, scroll bars, radio boxes, dials and buttons as necessary to develop new assessments of human behaviour.
Existing patents and scientific publications relating to tablet technology do not encompass the functionality of the fMRI-compatible tablet. Use of a resistive sensor, as proposed by Reithler et al. in their 2004 publication, limits researchers to a small subset of experiments and does not allow for natural capture of handwriting or drawing during fMRI. Re-emphasizing, it would be very advantageous to provide a device to record and visualize drawing and writing movements during fMRI that would allow individuals lying inside the scanner to interact with the device in a natural, intuitive way, similar to writing with pen and paper, and that would significantly augment current fMRI applications. The invention described here fulfills this need by allowing drawing movements to be captured and recorded during fMRI with optional visual feedback relayed to the user.